Medical image processing apparatus, medical image processing system, medical image processing method, and program

ABSTRACT

There are provided a medical image processing apparatus, a medical image processing system, a medical image processing method, and a program that allow an observer to grasp a region that contributes to classification in automatic classification of a medical image. A medical image processing apparatus includes an image acquisition unit ( 40 ) that acquires a captured image ( 38 ) generated through imaging of a living body; a classification unit ( 48 ) that classifies the captured image into two or more classes; an image generation unit ( 50 ) that generates a region image depicting a location of a region that contributes to classification performed using the classification unit in the captured image subjected to the classification performed using the classification unit; and a display signal transmission unit ( 44 ) that transmits, to a display device, a display signal representing the captured image, a classification result derived using the classification unit, and the region image. The display signal transmission unit transmits, to the display device, a display signal for displaying the region image separately from the captured image.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation of PCT InternationalApplication No. PCT/JP2019/023696 filed on Jun. 14, 2019 claimingpriority under 35 U.S.C § 119(a) to Japanese Patent Application No.2018-130138 filed on Jul. 9, 2018. Each of the above applications ishereby expressly incorporated by reference, in its entirety, into thepresent application.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a medical image processing apparatus, amedical image processing system, a medical image processing method, anda program, and more particularly to automatic classification of a lesionin a medical image.

2. Description of the Related Art

In the medical field, examinations using a modality such as an endoscopesystem are carried out. In recent years, there has been known atechnique of analyzing medical images such as endoscopic images whichare time-series images captured using an endoscope scope, automaticallyclassifying a lesion included in the endoscopic images, and providing aclassification result. Note that classification of images anddiscrimination of images are treated as the same concept herein.

JP5528255B describes an endoscopic image processing system that assistsdetection of a characteristic lesion site in an image captured with anendoscope. The endoscopic image processing system described inJP5528255B displays a dotted line so as to surround a location estimatedto be a lesion site in an image displayed on a monitor when a lesionestimation function is executed.

JP2017-70609A describes an image processing system that performs textureanalysis on the density of the entirety of a captured image of a bloodvessel, and uses the result of the texture analysis to performclassification corresponding to pathological diagnoses of a non-tumor,an adenoma, and the like. JP2017-70609A describes an example ofdisplaying a probability of the classification being correct.

JP4615963B describes an endoscope apparatus that removes an imageinappropriate for diagnosis from among images of the inside of a bodycavity captured using the endoscope apparatus.

JP5576711B describes an image processing apparatus that processestime-series images of an in-vivo lumen captured in time series using amedical observation apparatus. The image processing apparatus describedin JP5576711B determines, as a specific region, a region of normalmucosa in an image constituting the time-series images. The imageprocessing apparatus described in JP5576711B also calculates a degree ofreliability of the specific region.

SUMMARY OF THE INVENTION

When the type of a lesion found in an examination of the inside of abody cavity performed using an endoscope apparatus is automaticallyclassified using artificial intelligence (AI), the AI that performsautomatic classification receives images acquired during the examinationand outputs a classification result.

However, the images acquired during the examination include images thatare not suitable for classification using the AI, such as an image inwhich a part of a lesion is not depicted because the lesion is hidden byan object although the lesion is present and an image in which a lesionis depicted but is blurred. In such a case, the AI may output aninappropriate classification result.

With the inventions described in JP5528255B, JP2017-70609A, JP4615963B,and JP5576711B, when an image that is not suitable for automaticclassification described above is classified, an inappropriateclassification result may be output. On the other hand, it is difficultfor an observer to determine whether the classification result isappropriate.

The present invention has been made in view of such circumstances, andan object of the present invention is to provide a medical imageprocessing apparatus, a medical image processing system, a medical imageprocessing method, and a program that allow an observer to grasp whetherclassification result is appropriate in automatic classification of amedical image.

In order to accomplish the object described above, the following aspectsof the invention are provided.

A medical image processing apparatus according to a first aspectincludes: an image acquisition unit that acquires a captured imagegenerated through imaging of a living body; a classification unit thatclassifies the captured image into two or more classes; an imagegeneration unit that generates a region image depicting a location of aregion that contributes to classification performed using theclassification unit, in the captured image subjected to theclassification performed using the classification unit; and a displaysignal transmission unit that transmits to a display device, a displaysignal representing the captured image, a classification result derivedusing the classification unit, and the region image. The display signaltransmission unit transmits, to the display device, a display signal fordisplaying the region image separately from the captured image.

According to the first aspect, the captured image is classified into twoor more classes, the region image depicting the region that contributesto the classification in the captured image is generated, and the regionimage is displayed separately from the captured image using the displaydevice. This thus allows an observer to grasp a region that contributesto classification and to grasp whether the classification isappropriate.

A medical image captured using medical equipment may be used as thecaptured image. The medical image refers to a captured image of a livingbody generated using a modality such as an endoscope apparatus, acomputed tomography (CT) apparatus, a magnetic resonance imaging (MRI)apparatus, or an X-ray imaging apparatus.

Predetermined medical classifications may be used as the classes.

A second aspect may be configured such that in the medical imageprocessing apparatus according to the first aspect, the image generationunit changes a depicting manner of the region image in accordance withthe classification result of the captured image.

According to the second aspect, it becomes easier to recognize thedistinction between the classifications.

Examples of the depicting manner include a depicting manner using colorand a depicting manner in which the location is changed. A depictingmanner using both of these depicting manners may also be used.

A third aspect may be configured such that in the medical imageprocessing apparatus according to the first or second aspect, theclassification unit classifies the captured image on the basis of afeature quantity acquired from the captured image, and the imagegeneration unit generates the region image on the basis of the featurequantity.

According to the third aspect, the region image based on the featurequantity of the captured image can be generated.

In the third aspect, the captured image may be divided into a pluralityof regions, a feature quantity may be calculated for each of theregions, and a region that contributes to classification in the capturedimage may be identified on the basis of the features of the respectiveregions.

A fourth aspect may be configured such that in the medical imageprocessing apparatus according to the first or second aspect, theclassification unit employs a deep learning device that has beentrained, and the image generation unit generates the region image on thebasis of information of an intermediate layer of the deep learningdevice.

According to the fourth aspect, the region image based on theinformation of the intermediate layer of the trained deep learningdevice can be generated.

A fifth aspect may be configured such that in the medical imageprocessing apparatus according to any one of the first to fourthaspects, the classification unit calculates, for each of a plurality ofregions set in the captured image, membership degrees for the classes,and classifies the captured image on the basis of the membershipdegrees.

According to the fifth aspect, classification based on the membershipdegrees can be performed.

Examples of the membership degrees include membership probabilities forthe classes and scores for the classes.

A sixth aspect may be configured such that in the medical imageprocessing apparatus according to the fifth aspect, the image generationunit generates the region image on the basis of the membership degrees.

According to the sixth aspect, the region image based on the membershipdegrees can be generated.

A seventh aspect may be configured such that in the medical imageprocessing apparatus according to any one of the first to sixth aspects,the classification unit performs exception determination for theclassification based on the region image, and the display signaltransmission unit transmits, to the display device, a display signalrepresenting a result of the exception determination performed using theclassification unit.

According to the seventh aspect, output of an inappropriateclassification result may be suppressed for a captured image that isdifficult to classify.

The display signal transmission unit may transmit, to the displaydevice, a display signal representing a result of the exceptiondetermination instead of the classification result. The display signaltransmission unit may transmit, to the display device, a display signalrepresenting the classification result and a result of the exceptiondetermination.

An eighth aspect may be configured such that in the medical imageprocessing apparatus according to the seventh aspect, the classificationunit calculates, on the basis of the region image, a degree ofreliability of the classification result derived using theclassification unit, and the display signal transmission unit transmits,to the display device, a display signal representing the degree ofreliability.

According to the eighth aspect, the degree of reliability ofclassification may be grasped.

A ninth aspect may be configured such that the medical image processingapparatus according to the eighth aspect further includes: a storageinstruction acquisition unit that acquires an instruction to store thecaptured image; and a storage unit that stores the captured image. Thestorage unit associates, in storing the captured image in the storageunit in accordance with the instruction to store the captured image, atleast any of the classification result, the result of the exceptiondetermination, or the degree of reliability of the classification resultwith the captured image.

According to the ninth aspect, the captured image and the informationassociated with the captured image may be used. In addition, theinformation associated with the captured image may be checked.

A tenth aspect may be configured such that in the medical imageprocessing apparatus according to any one of the first to ninth aspects,the display signal transmission unit transmits to the display device, adisplay signal representing text information of the classificationresult.

According to the tenth aspect, the classification result may be graspedon the basis of the text information.

The text information may be in a language of any kind. An abbreviationmay be used as the text information.

A medical image processing system according to an eleventh aspectincludes: an image acquisition unit that acquires a captured imagegenerated through imaging of a living body; a classification unit thatclassifies the captured image into two or more classes; an imagegeneration unit that generates a region image depicting a location of aregion that contributes to classification performed using theclassification unit, in the captured image subjected to theclassification performed using the classification unit; a display devicethat displays the captured image and the region image, and a displaysignal transmission unit that transmits, to the display device, adisplay signal representing the captured image and the region image. Thedisplay signal transmission unit transmits, to the display device, adisplay signal for displaying the region image separately from thecaptured image.

According to the eleventh aspect, substantially the same advantages asthose of the first aspect can be obtained.

The eleventh aspect may be appropriately combined with any of featuresthat are substantially the same as those specified in the second totenth aspects. In such a case, a constituent element responsible for aprocess or function specified in the medical image processing apparatuscan be grasped as a constituent element responsible for thecorresponding process or function in the medical image processingsystem.

A medical image processing method according to a twelfth aspectincludes: an image acquisition step of acquiring a captured imagegenerated through imaging of a living body; a classification step ofclassifying the captured image into two or more classes; an imagegeneration step of generating a region image depicting a location of aregion that contributes to classification performed in theclassification step, in the captured image subjected to theclassification performed in the classification step; and a displaysignal transmission step of transmitting, to a display device, a displaysignal representing the captured image, a classification result derivedin the classification step, and the region image. The display signaltransmission step transmits, to the display device, a display signal fordisplaying the region image separately from the captured image.

According to the twelfth aspect, substantially the same advantages asthose of the first aspect can be obtained.

The twelfth aspect may be appropriately combined with any of featuresthat are substantially the same as those specified in the second totenth aspects. In such a case, a constituent element responsible for aprocess or function specified in the medical image processing apparatuscan be grasped as a constituent element responsible for thecorresponding process or function in the medical image processingmethod.

A program according to a thirteenth aspect causes a computer toimplement: an image acquisition function that acquires a captured imagegenerated through imaging of a living body; a classification functionthat classifies the captured image into two or more classes; an imagegeneration function that generates a region image depicting a locationof a region that contributes to classification performed using theclassification function, in the captured image subjected to theclassification performed using the classification function; and adisplay signal transmission function that transmits, to a displaydevice, a display signal representing the captured image, aclassification result derived using the classification function, and theregion image. The display signal transmission function is configured totransmit, to the display device, a display signal for displaying theregion image separately from the captured image.

According to the thirteenth aspect, substantially the same advantages asthose of the first aspect can be obtained.

The thirteenth aspect may be appropriately combined with any of featuresthat are substantially the same as those specified in the second totenth aspects. In such a case, a constituent element responsible for aprocess or function specified in the medical image processing apparatuscan be grasped as a constituent element responsible for thecorresponding process or function in the program.

According to the present invention, the captured image is classifiedinto two or more classes, the region image depicting the region thatcontributes to classification in the captured image is generated, andthe region image is displayed separately from the captured image usingthe display device. This thus allows an observer to grasp a region thatcontributes to classification and to grasp whether the classification isappropriate.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an overall configuration diagram of an endoscope systemincluding a medical image processing apparatus according to embodiments;

FIG. 2 is a block diagram illustrating a hardware configuration of themedical image processing apparatus;

FIG. 3 is a functional block diagram of the medical image processingapparatus according to a first embodiment;

FIG. 4 is an explanatory diagram of a display screen used in the firstembodiment;

FIG. 5 is an explanatory diagram of a display screen according to acomparative example;

FIG. 6 is an explanatory diagram of a display screen displayed when thereliability of a classification result is low;

FIG. 7 is an explanatory diagram of a display screen of another exampledisplayed when the reliability of the classification result is low;

FIG. 8 is an explanatory diagram of a display screen displayed when aplurality of lesions are present;

FIG. 9 is an explanatory diagram of a display example of aclassification result;

FIG. 10 is an explanatory diagram of an example of displaying membershipprobabilities as classification results;

FIG. 11 is an explanatory diagram of an example of displaying scores asclassification results;

FIG. 12 is a flowchart illustrating a procedure of a medical imageprocessing method;

FIG. 13 is an explanatory diagram of a display screen according to amodification of a region image;

FIG. 14 is an explanatory diagram of a display screen according to afirst modification of the classification result;

FIG. 15 is an explanatory diagram of a display screen according to asecond modification of the classification result;

FIG. 16 is an explanatory diagram of a region image to which densityaccording to a degree of contribution is applied;

FIG. 17 is an explanatory diagram of a region image to which a heat mapaccording to the degree of contribution is applied;

FIG. 18 is a block diagram illustrating an example of a configuration ofa classification unit that employs a convolutional neural network;

FIG. 19 is a conceptual diagram of the shape transition of a featurequantity in the convolutional neural network;

FIG. 20 is a conceptual diagram of depiction based on information of anintermediate layer of the convolutional neural network;

FIG. 21 is an explanatory diagram of a segmentation technique;

FIG. 22 is an explanatory diagram of a display screen indicating theclassification result of “undetectable”;

FIG. 23 is an explanatory diagram of a display screen indicating theclassification result of “undeterminable”; and

FIG. 24 is a functional block diagram of a medical image processingapparatus according to a third embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Preferred embodiments of the present invention will be described indetail below in accordance with the accompanying drawings. The sameconstituent elements are denoted by the same reference signs herein, andredundant description will be appropriately omitted.

Overall Configuration of Endoscope System

FIG. 1 is an overall configuration diagram of an endoscope systemincluding a medical image processing apparatus according to embodiments.An endoscope system 9 illustrated in FIG. 1 includes an endoscope 10, alight source device 11, a processor device 12, a display device 13, amedical image processing apparatus 14, an input device 15, and a monitordevice 16.

The endoscope 10 is an electronic endoscope. The endoscope 10 is also aflexible endoscope. The endoscope 10 includes an insertion section 20,an operation section 21, and a universal cord 22. The insertion section20 is inserted into a subject. The entire insertion section 20 is formedto have an elongated shape with a small diameter.

The insertion section 20 includes a soft part 25, a bending part 26, anda tip part 27. The soft part 25, the bending part 26, and the tip part27 are coupled to each other to constitute the insertion section 20. Thesoft part 25 has flexibility sequentially from a proximal end side to adistal end side of the insertion section 20. The bending part 26 has astructure that is bendable when the operation section 21 is operated.The tip part 27 includes an imaging optical system (not illustrated), animaging element 28, and so on.

A CMOS imaging element or a CCD imaging element is used as the imagingelement 28. CMOS is an abbreviation for complementary metal oxidesemiconductor. CCD is an abbreviation for charge coupled device.

An observation window (not illustrated) is disposed on a tip surface 27a of the tip part 27. The observation window is an opening formed on thetip surface 27 a of the tip part 27. A cover (not illustrated) isattached to the observation window. The imaging optical system (notillustrated) is disposed behind the observation window. Image light of asite to be observed is incident onto an imaging surface of the imagingelement 28 through the observation window, the imaging optical system,and so on. The imaging element 28 images the image light of the site tobe observed incident onto the imaging surface of the imaging element 28and outputs an imaging signal. The term “imaging” used herein includesthe meaning of converting light reflected off from a site to be observedinto an electric signal.

The operation section 21 is coupled to the proximal end side of theinsertion section 20. The operation section 21 includes variousoperating members to be operated by a technician. Specifically, theoperation section 21 includes two types of bending operation knobs 29.The bending operation knobs 29 are used to perform an operation ofbending the bending part 26. Note that the technician may also bereferred to as a doctor, an operator, an observer, a user, or the like.

The operation section 21 includes an air/water supply button 30 and asuction button 31. The air/water supply button 30 is used when thetechnician performs an air/water supply operation. The suction button 31is used when the technician performs a suction operation.

The operation section 21 includes a still image capturing instructionpart 32 and a treatment tool introduction port 33. The still imagecapturing instruction part 32 is operated by the technician when a stillimage of the site to be observed is captured. The treatment toolintroduction port 33 is an opening through which a treatment tool is tobe inserted into a treatment tool insertion path that is inserted insidethe insertion section 20. Note that illustration of the treatment toolinsertion path and the treatment tool is omitted. A still image,assigned a reference sign 39, is illustrated in FIG. 3.

The universal cord 22 is a connection cord that connects the endoscope10 to the light source device 11. The universal cord 22 includes thereina light guide 35, a signal cable 36, and a fluid tube (not illustrated),which are inserted inside the insertion section 20.

In addition, a tip part of the universal cord 22 includes a connector 37a to be connected to the light source device 11 and a connector 37 bbranching from the connector 37 a and to be connected to the processordevice 12.

When the connector 37 a is connected to the light source device 11, thelight guide 35 and the fluid tube (not illustrated) are inserted intothe light source device 11. Consequently, necessary illumination light,water, and gas are supplied from the light source device 11 to theendoscope 10 through the light guide 35 and the fluid tube (notillustrated).

As a result, the illumination light is radiated from an illuminationwindow (not illustrated) of the tip surface 27 a of the tip part 27toward the site to be observed. In addition, in response to an operationof pressing the air/water supply button 30, gas or water is ejected froman air/water supply nozzle (not illustrated) of the tip surface 27 a ofthe tip part 27 toward the observation window (not illustrated) of thetip surface 27 a.

When the connector 37 b is connected to the processor device 12, thesignal cable 36 and the processor device 12 are electrically connectedto each other. Consequently, an imaging signal of the site to beobserved is output from the imaging element 28 of the endoscope 10 tothe processor device 12 through the signal cable 36. Also, a controlsignal is output from the processor device 12 to the endoscope 10through the signal cable 36.

In the present embodiments, the flexible endoscope is described as anexample of the endoscope 10. However, various types of electronicendoscopes capable of capturing a moving image of a site to be observed,such as a rigid endoscope, may be used as the endoscope 10.

The light source device 11 supplies illumination light to the lightguide 35 of the endoscope 10 through the connector 37 a. White light orlight in a specific wavelength range is usable as the illuminationlight. The illumination light may be a combination of white light andlight in a specific wavelength range. The light source device 11 isconfigured to be able to appropriately select, as the illuminationlight, light in a wavelength range corresponding to an observationpurpose.

The white light may be light in a white wavelength range or light in aplurality of wavelength ranges. The specific wavelength range is a rangenarrower than the white wavelength range. As the light in the specificwavelength range, light in a single wavelength range may be used, orlight in a plurality of wavelength ranges may be used. The light in thespecific wavelength range may be referred to as special light.

The processor device 12 controls the operation of the endoscope 10through the connector 37 b and the signal cable 36. The processor device12 also acquires an imaging signal from the imaging element 28 of theendoscope 10 through the connector 37 b and the signal cable 36. Theprocessor device 12 uses a predetermined frame rate to acquire animaging signal output from the endoscope 10.

The processor device 12 generates an endoscopic image, which is anobservation image of the site to be observed, on the basis of theimaging signal acquired from the endoscope 10. Herein, an endoscopicimage 38 includes a moving image. The endoscopic image 38 may includethe still image 39. Note that a moving image, assigned a reference sign38 a, is illustrated in FIG. 3. The endoscopic image 38 described in theembodiments is an example of a captured image.

When the still image capturing instruction part 32 of the operationsection 21 is operated, the processor device 12 generates the stillimage 39 of the site to be observed on the basis of the imaging signalacquired from the imaging element 28 in parallel with generation of themoving image. The still image 39 may be generated to have a resolutionhigher than the resolution of the moving image.

When the endoscopic image 38 is generated, the processor device 12performs image quality correction in which digital signal processingsuch as white balance adjustment and shading correction are used. Theprocessor device 12 may add accessory information defined by the DICOMstandard to the endoscopic image 38. Note that DICOM is an abbreviationfor Digital Imaging and Communications in Medicine.

The endoscopic image 38 is an in-vivo image depicting the inside of asubject, that is, the inside of a living body. If the endoscopic image38 is an image obtained through imaging using light in a specificwavelength range, the endoscopic image 38 is a special-light image. Theprocessor device 12 then outputs the generated endoscopic image 38 toeach of the display device 13 and the medical image processing apparatus14. The processor device 12 may output the endoscopic image 38 to astorage device (not illustrated) via a network (not illustrated) inaccordance with a communication protocol compliant with the DICOMstandard. Note that a network 140 illustrated in FIG. 2 may be used asthe network.

The display device 13 is connected to the processor device 12. Thedisplay device 13 displays the endoscopic image 38 transmitted from theprocessor device 12. The technician may perform an operation of movingthe insertion section 20 forward and backward while checking theendoscopic image 38 displayed on the display device 13. Upon detecting alesion or the like at the site to be observed, the technician mayoperate the still image capturing instruction part 32 to capture a stillimage of the site to be observed.

A computer is used as the medical image processing apparatus 14. Akeyboard, a mouse, and the like connectable to the computer are used asthe input device 15. The input device 15 and the computer may beconnected to each other either with a cable or wirelessly. Variousmonitors connectable to the computer are used as the monitor device 16.

As the medical image processing apparatus 14, a diagnosis assistantapparatus such as a workstation or a server apparatus may be used. Inthis case, the input device 15 and the monitor device 16 are providedfor each of a plurality of terminals connected to the workstation or thelike. Further, as the medical image processing apparatus 14, a medicalservice assistant apparatus that assists creation of a medical report orthe like may be used.

The medical image processing apparatus 14 acquires the endoscopic image38 and stores the endoscopic image 38. The medical image processingapparatus 14 controls reproduction performed by the monitor device 16.Note that the term “image” used herein includes a meaning of an electricsignal representing the image and a meaning of image data such asinformation representing the image. The term “image” used herein meansat least any of an image itself or image data.

Further, the term “storing an image” can be interpreted as “saving animage”. “Storing an image” used herein means “storing an image in anon-transitory manner”. The medical image processing apparatus 14 mayinclude a temporary storage memory that temporarily stores an image.

The input device 15 is used to input an operation instruction for themedical image processing apparatus 14. The monitor device 16 displaysthe endoscopic image 38 under the control of the medical imageprocessing apparatus 14. The monitor device 16 may function as a displaydevice of various kinds of information in the medical image processingapparatus 14.

The medical image processing apparatus 14 may be connected to a storagedevice (not illustrated) via a network (not illustrated in FIG. 1). TheDICOM standard, a protocol compliant with the DICOM standard, and thelike may be used as the image storage format and for the communicationbetween apparatuses via the network.

As the storage device (not illustrated), a storage or the like thatstores data in a non-transitory manner may be used. The storage devicemay be managed using a server apparatus (not illustrated). As the serverapparatus, a computer that stores and manages various kinds of data maybe used.

A configuration including the medical image processing apparatus 14 andthe monitor device 16 described in the embodiments is an example of amedical image processing system.

Description of Medical Image Processing Apparatus According to FirstEmbodiment Hardware Configuration of Medical Image Processing Apparatus

FIG. 2 is a block diagram illustrating a hardware configuration of themedical image processing apparatus. The medical image processingapparatus 14 illustrated in FIG. 2 includes a control unit 120, a memory122, a storage device 124, a network controller 126, a power supplydevice 128, a display controller 130, an input/output interface 132, andan input controller 134. Note that I/O illustrated in FIG. 2 representsthe input/output interface 132.

The control unit 120, the memory 122, the storage device 124, thenetwork controller 126, the display controller 130, and the input/outputinterface 132 are connected to each other via a bus 136 so that datacommunication can be performed therebetween.

Control Unit

The control unit 120 functions as an overall control unit, variouscalculation units, and a storage control unit of the medical imageprocessing apparatus 14. The control unit 120 executes a program storedin a read-only memory (ROM) included in the memory 122.

The control unit 120 may download a program from an external storagedevice (not illustrated) via the network controller 126 and execute thedownloaded program. The external storage device may be communicablyconnected to the medical image processing apparatus 14 via the network140.

The control unit 120 uses, as a calculation area, a random access memory(RAM) included in the memory 122 and executes various processes incooperation with various programs. Consequently, various functions ofthe medical image processing apparatus 14 are implemented.

The control unit 120 controls reading out of data from the storagedevice 124 and writing of data to the storage device 124. The controlunit 120 may acquire various kinds of data from an external storagedevice via the network controller 126. The control unit 120 is capableof executing various processes such as calculations using the acquiredvarious kinds of data.

The control unit 120 may include one processor or two or moreprocessors. Examples of the processor include a field programmable gatearray (FPGA), a programmable logic device (PLD), and so on. An FPGA anda PLD are devices whose circuit configurations are changeable afterbeing manufactured.

Another example of the processor is an application-specific integratedcircuit (ASIC). An ASIC includes a circuit configuration dedicatedlydesigned to execute specific processing.

The control unit 120 may use two or more processors of the same kind.For example, the control unit 120 may use two or more FPGAs or two ormore PLDs. The control unit 120 may use two or more processors ofdifferent kinds. For example, the control unit 120 may use one or moreFPGAs and one or more ASICs.

When the medical image processing apparatus 14 includes a plurality ofcontrol units 120, the plurality of control units 120 may be configuredusing a single processor. As an example of configuring the plurality ofcontrol units 120 using a single processor, there is a form in which thesingle processor is configured using a combination of one or morecentral processing units (CPUs) and software and this processorfunctions as the plurality of control units 120. Note that software usedherein is synonymous with a program.

As another example of configuring the plurality of control units 120using a single processor, there is a form in which a processor thatimplements, with a single IC chip, the functions of the entire systemincluding the plurality of control units 120. Representative examples ofthe processor that implements, with a single IC chip, the functions ofthe entire system including the plurality of control units 120 include asystem on a chip (SoC). Note that IC is an abbreviation for integratedcircuit.

As described above, the control unit 120 is configured using one or moreof various kinds of processors as the hardware structure.

Memory

The memory 122 includes a ROM (not illustrated) and a RAM (notillustrated). The ROM stores various programs to be executed in themedical image processing apparatus 14. The ROM stores parameters, files,and the like used for executing various programs. The RAM functions as atemporary data storage area, a work area for the control unit 120, andthe like.

Storage Device

The storage device 124 stores various kinds of data in a non-transitorymanner. The storage device 124 may be externally attached to the medicalimage processing apparatus 14. Instead of or along with the storagedevice 124, a large-capacity semiconductor memory device may be used.

Network Controller

The network controller 126 controls data communication between themedical image processing apparatus 14 and an external apparatus. Thecontrol of the data communication may include management of the trafficin the data communication. As the network 140 to which the medical imageprocessing apparatus 14 is connected via the network controller 126, aknown network such as a local area network (LAN) may be used.

Power Supply Device

As the power supply device 128, a large-capacity power supply devicesuch as an uninterruptible power supply (UPS) is used. The power supplydevice 128 supplies power to each unit of the medical image processingapparatus 14 when the commercial power supply is cut off due to a powerfailure or the like.

Display Controller

The display controller 130 functions as a display driver that controlsthe monitor device 16 in accordance with a command signal transmittedfrom the control unit 120.

Input/Output Interface

The input/output interface 132 communicably connects the medical imageprocessing apparatus 14 and an external device to each other. Acommunication standard such as Universal Serial Bus (USB) may be usedfor the input/output interface 132.

Input Controller

The input controller 134 converts the format of a signal input using theinput device 15 into a format suitable for processing performed by themedical image processing apparatus 14. Information input from the inputdevice 15 via the input controller 134 is transmitted to each unit viathe control unit 120.

Note that the hardware configuration of the medical image processingapparatus 14 illustrated in FIG. 2 is merely an example. Thus, addition,deletion, and modification may be appropriately made. Note that thehardware configuration of the medical image processing apparatus 14illustrated in FIG. 2 is also applicable to embodiments other than afirst embodiment.

Description of Functional Blocks of Medical Image Processing Apparatus

FIG. 3 is a functional block diagram of the medical image processingapparatus according to the first embodiment. The medical imageprocessing apparatus 14 includes an image acquisition unit 40, an imageprocessing unit 42, a display control unit 44, and a storage unit 46.The image acquisition unit 40 acquires the endoscopic image 38 from theprocessor device 12. The image acquisition unit 40 stores the endoscopicimage 38 in an endoscopic image storage unit 46 a.

The image acquisition unit 40 may acquire the endoscopic image 38 fromthe processor device 12 via an information storage medium such as amemory card. The image acquisition unit 40 may acquire the endoscopicimage 38 via the network 140 illustrated in FIG. 2.

That is, the image acquisition unit 40 may acquire the moving image 38 aconstituted by time-series frame images 38 b. The image acquisition unit40 may acquire the still image 39 in the case where still imagecapturing is performed during capturing of the moving image 38 a.

The image processing unit 42 includes a classification unit 48 and aregion image generation unit 50. The classification unit 48 performsautomatic classification of a lesion from the endoscopic image 38. Theterm “classification” used herein can be read as “discrimination”.

Specifically, the classification unit 48 may classify the endoscopicimage 38 into a predetermined class and derive a classification result.The classification unit 48 adds class information to the frame images 38b constituting the moving image 38 a at the time of automaticclassification of the endoscopic image 38. The classification unit 48may add the class information to all the frame images 38 b, or may addthe class information to the frame images 38 b of every several frames.The classification unit 48 may add the class information to the stillimage 39. The class and the class information can be read as aclassification result.

The classification unit 48 stores the classification result in aclassification storage unit 46 b in association with the frame images 38b. Table 1 below illustrates examples of classes used by theclassification unit 48.

TABLE 1 Class Specific example Tumor or non-tumor Tumor, Non-tumorClassification of NICE classification, JNET endoscopic findingsclassification, etc. Type Hyperplastic polyp, Adenoma, Intramucosalcarcinoma, Highly invasive carcinoma, Inflammatory polyp, etc. Note thatNICE in Table 1 above is an abbreviation for NBI InternationalColorectal Endoscopic Classification. NBI is an abbreviation for NarrowBand Imaging. JNET is an abbreviation for The Japan NBI Expert Team.

The region image generation unit 50 generates a region imagerepresenting a region that contributes to classification in theendoscopic image 38. The region image generation unit 50 stores theregion image in a region image storage unit 46 c. The region image,assigned a reference sign 208, is illustrated in FIG. 4.

The display control unit 44 transmits, to the monitor device 16, adisplay control signal that causes the monitor device 16 to display theendoscopic image 38 and the region image generated using the regionimage generation unit 50. The display control unit 44 transmits, to themonitor device 16, a display control signal that causes the monitordevice 16 to display text information representing the classificationresult of the endoscopic image 38 derived using the classification unit48. The display control unit 44 described in the embodiments is anexample of a display signal transmission unit.

The monitor device 16 displays the endoscopic image 38, the regionimage, and the text information representing the classification resultin the same screen. The monitor device 16 displays the endoscopic image38 and the region image in different regions in the screen. Textinformation, assigned a reference sign 212, representing theclassification result is illustrated in FIG. 4. Details of the screendisplayed using the monitor device 16 will be described later.

The storage unit 46 includes the endoscopic image storage unit 46 a, theclassification storage unit 46 b, and the region image storage unit 46c. The endoscopic image storage unit 46 a stores the endoscopic image 38acquired using the image acquisition unit 40.

The image processing unit 42 reads out the endoscopic image 38 stored inthe endoscopic image storage unit 46 a and performs image processing onthe endoscopic image 38. The display control unit 44 reads out theendoscopic image 38 stored in the endoscopic image storage unit 46 a andcauses the monitor device 16 to display the endoscopic image 38.

The classification storage unit 46 b stores the class of the endoscopicimage 38 classified using the classification unit 48 in association withthe endoscopic image 38. Specifically, the classification storage unit46 b stores, in association with the frame image 38 b, the class of theframe image 38 b constituting the moving image 38 a. The display controlunit 44 reads out the classification result from the classificationstorage unit 46 b and causes the monitor device 16 to display the textinformation or the like representing the classification result.

The region image storage unit 46 c stores the region image generatedusing the region image generation unit 50. The display control unit 44reads out the region image from the region image storage unit 46 c andcauses the monitor device 16 to display the region image.

One or more storage elements may be used as the storage unit 46illustrated in FIG. 3. That is, the storage unit 46 may include threestorage elements corresponding to the endoscopic image storage unit 46a, the classification storage unit 46 b, and the region image storageunit 46 c, respectively. A plurality of storage elements may be used aseach of the endoscopic image storage unit 46 a, the classificationstorage unit 46 b, and the region image storage unit 46 c. Furthermore,two or all of the endoscopic image storage unit 46 a, the classificationstorage unit 46 b, and the region image storage unit 46 c may beconfigured using a single storage element.

Description of Display Screen Displayed on Monitor Device Description ofRegion Image

FIG. 4 is an explanatory diagram of a display screen used in the firstembodiment. A display screen 200 illustrated in FIG. 4 includes anendoscopic image display area 202, a region image display area 204, anda classification result display area 206.

The endoscopic image display area 202 is an area in which the endoscopicimage 38 is displayed. The still image 39 may be displayed in theendoscopic image display area 202. The endoscopic image 38 and the stillimage 39 may be displayed in a switching manner in the endoscopic imagedisplay area 202. A reference sign 209 denotes a classificationcontribution region 209 which is a region that contributes toclassification of the endoscopic image 38. FIG. 4 schematicallyillustrates the classification contribution region 209.

The region image 208 is displayed in the region image display area 204.In the region image 208, a classification contribution correspondingregion 210 is displayed with highlight. The classification contributioncorresponding region 210 is a region, in the region image 208,corresponding to the classification contribution region 209.

The text information 212 representing the classification result isdisplayed in the classification result display area 206. FIG. 4illustrates an example in which “neoplastic”, which is the Englishnotation for a tumor, is displayed as the text information 212 in theclassification result display area 206. Note that the text information212 may be in a language of any kind. That is, Japanese notation orforeign language notation other than English may be used for the textinformation 212. An abbreviation may also be used for the textinformation 212.

FIG. 5 is an explanatory diagram of a display screen according to acomparative example. A comparative screen 220 displays the result ofautomatic classification, and indicates an example in which a tumor 222is found from the endoscopic image 38 and the endoscopic image 38 isclassified to the tumor. The endoscopic image 38 and the textinformation 212 representing the classification result are displayed inthe comparative screen 220. Note that FIG. 5 schematically illustratesthe tumor 222.

The text information 212 representing the classification result isdisplayed in the comparative screen 220 illustrated in FIG. 5. However,when the endoscopic image 38 is difficult to classify, an incorrectclassification result may be output. On the other hand, the region image208 corresponding to the endoscopic image 38 is displayed in the displayscreen 200 illustrated in FIG. 4, and the classification contributioncorresponding region 210 is displayed in the region image 208. Thisallows an observer to visually grasp which region in the endoscopicimage 38 the classification unit 48 performs classification on the basisof. In addition, the region image 208 may serve as an index of thereliability of the classification result.

FIG. 6 is an explanatory diagram of a display screen displayed when thereliability of the classification result is low. A display screen 200 aillustrated in FIG. 6 corresponds to an example of the case where theclassification result is incorrect and the classification is performedon the basis of a region different from the tumor 222 in the endoscopicimage 38.

In a region image 208 a illustrated in FIG. 6, a region 230 differentfrom a region to be set as a classification contribution correspondingregion 210 a is displayed as the classification contributioncorresponding region.

FIG. 7 is an explanatory diagram of a display screen of another exampledisplayed when the reliability of the classification result is low. Adisplay screen 200 b illustrated in FIG. 7 corresponds to an exampledisplayed in the case where the tumor 222 in the endoscopic image 38fails to be found.

In a region image 208 b illustrated in FIG. 7, a region 210 b to be setas the classification contribution corresponding region is notdisplayed. When the display screen 200 b illustrated in FIG. 7 isdisplayed, it is considered that the classification unit 48 fails tofind a target to be classified due to reasons such as the tumor 222being blurred and the size of the tumor 222 being small.

In such a case, the operator operates the endoscope 10 illustrated inFIG. 1 to adjust the focus or to display the lesion and a peripheralregion of the lesion in an enlarged manner. This allows theclassification unit 48 to perform classification correctly. That is,displaying the region image 208 together with the endoscopic image 38may serve as a suggestion that prompts the operator to perform anoperation that causes the classification unit 48 to derive the correctclassification result.

In the present embodiment, the example in which the endoscopic imagedisplay area 202 and the region image display area 204 are displayed inthe display screen 200 displayed on the single monitor device 16 hasbeen described. Alternatively, the display screen 200 including theendoscopic image display area 202 may be displayed on one of two monitordevices, and the display screen 200 including the region image displayarea 204 may be displayed on the other monitor device.

In addition, the endoscopic image display area 202 and the region imagedisplay area 204 may be displayed in the single display screen 200displayed on the single monitor device 16 so as to be switched in a timedivision manner. Further, the region image display area 204 may bedisplayed to be superimposed on the endoscopic image display area 202.For example, the region image display area 204 may be displayed to besuperimposed, at a location where observation of the endoscopic image 38is not hindered in the endoscopic image display area 202, such as alower left corner of the endoscopic image display area 202 illustratedin FIG. 7.

The observer tends to dislike movement of the viewpoint because theobserver observes the endoscopic image while performing a preciseoperation. When the region image display area 204 is displayed to besuperimposed on the endoscopic image display area 202, the endoscopicimage display area 202 and the region image display area 204 in thedisplay screen 200 b are arranged at closer locations. This effectivelyreduces movement of the viewpoint of the observer.

FIG. 8 is an explanatory diagram of a display screen displayed when aplurality of lesions are present. In a display screen 200 d illustratedin FIG. 8, a first classification contribution region 209 a and a secondclassification contribution region 209 b are extracted for the pluralityof lesions that are present in the endoscopic image 38.

In a region image 208 g, a first classification contributioncorresponding region 210 g corresponding to the first classificationcontribution region 209 a and a second classification contributioncorresponding region 210 h corresponding to the second classificationcontribution region 209 b are displayed. A display manner is used thatenables the first classification contribution corresponding region 210 gand the second classification contribution corresponding region 210 h tobe distinguished from each other.

Further, in a classification result display area 206 a of the displayscreen 200 d, first text information 212 e representing a firstclassification result for the first classification contributioncorresponding region 210 g and second text information 212 frepresenting a second classification result for the secondclassification contribution corresponding region 210 h are displayed.

When lesions of different classifications, such as a non-neoplasticlesion and a neoplastic lesion, are present, it is difficult for asystem that outputs one classification result from the endoscopic image38 to return an appropriate classification result. In contrast, themedical image processing apparatus 14 according to the presentembodiment uses the display manners for the respective classificationsin the region image 208 g as illustrated in FIG. 8 to depict the firstclassification contribution region 209 a and the second classificationcontribution region 209 b. This enables appropriate classificationresults to be obtained even when a plurality of lesions are present inthe endoscopic image 38 and the classifications of the plurality oflesions are different from each other.

Description of Display of Classification Result

FIG. 9 is an explanatory diagram of a display example of aclassification result. FIG. 9 illustrates an example in which a specificclass is displayed as a classification result. Text information 212 aillustrated in FIG. 9 indicates that the classification result is NICE1.

FIG. 10 is an explanatory diagram of an example of displaying membershipprobabilities as the classification results. FIG. 10 illustrates anexample in which the membership probabilities for the respective classesare displayed as the classification results. Text information 212 billustrated in FIG. 10 indicates that the membership probability forNICE 1 is 98%, the membership probability for NICE 2 is 2%, and themembership probability for NICE 3 is 0%.

The text information 212 b illustrated in FIG. 10 may be text indicatingthat the membership probability for NICE 1 is 98% alone. The textinformation 212 b illustrated in FIG. 10 may be text indicating that themembership probability for NICE 1 is 98% and the membership probabilityfor NICE 2 is 2%.

FIG. 11 is an explanatory diagram of an example of displaying scores asthe classification results. FIG. 11 illustrates an example in which thescores for the respective classes are displayed as the classificationresults. Text information 212 c illustrated in FIG. 11 indicates thatthe score for NICE 1 is 1.23, the score for NICE 2 is 0.002, and thescore for NICE 3 is 0.05. The membership probabilities illustrated inFIG. 10 and the scores illustrated in FIG. 11 are examples of membershipdegrees for classes.

Procedure of Medical Image Processing Method

FIG. 12 is a flowchart illustrating a procedure of a medical imageprocessing method. In an endoscopic image acquisition step S10, theimage acquisition unit 40 illustrated in FIG. 3 acquires the endoscopicimage 38. In an endoscopic image storage step S12, the image acquisitionunit 40 stores the endoscopic image 38 acquired in the endoscopic imageacquisition step S10 in the endoscopic image storage unit 46 a.

In a classification step S14, the classification unit 48 classifies theendoscopic image 38 into a predetermined class. In a classificationresult storage step S16, the classification unit 48 stores theclassification result derived in the classification step S14 in theclassification storage unit 46 b.

In a region image generation step S18, the region image generation unit50 generates a region image such as the region image 208 illustrated inFIG. 4 on the basis of the classification result. In a region imagestorage step S20, the region image generation unit 50 stores the regionimage generated in the region image generation step S18 in the regionimage storage unit 46 c.

In a display signal transmission step S22, the display control unit 44transmits a display signal to the monitor device 16. The display signaltransmitted from the display control unit 44 to the monitor device 16includes a display signal representing the endoscopic image 38 and theregion image 208. The display signal transmitted from the displaycontrol unit 44 to the monitor device 16 may include a display signalrepresenting the classification result.

Modifications of Display Screen Modification of Region Image

FIG. 13 is an explanatory diagram of a display screen according to amodification of a region image. In a display screen 200 c illustrated inFIG. 13, a reduced-size image of the endoscopic image 38 is combined inthe background of a region image 208 c. That is, in the region image 208c, the classification contribution corresponding region 210 is displayedto be superimposed on the reduced-size endoscopic image 38. Thereduced-size image of the endoscopic image 38 may have a lowerresolution than the endoscopic image 38.

Modifications of Classification Result

FIG. 14 is an explanatory diagram of a display screen according to afirst modification of the classification result. In a region image 208 dillustrated in FIG. 14, text information 212 d representing theclassification result is displayed to be superimposed. In the regionimage 208 d, emphasis on the text information 212 d such as changing thecolor of the text information 212 d from the color of the classificationcontribution corresponding region 210 may be used.

FIG. 15 is an explanatory diagram of a display screen according to asecond modification of the classification result. In a display screen200 e illustrated in FIG. 15, a frame 201 of the endoscopic image 38 anda frame 207 of a region image 208 h are colored in accordance with theclassification result. Note that the frame 201 alone may be colored, orthe frame 207 alone may be colored. That is, in the display screen 200e, at least any of the frame 201 of the endoscopic image 38 or the frame207 of the region image 208 h is colored in accordance with theclassification result.

Since the observer observes the endoscopic image while performing aprecise operation, there is a concern that the movement of the viewpointof the observer, the visual recognition of the text information by theobserver, and the like may adversely affect the operation performed bythe observer. In contrast, the operator who views the display screen 200e illustrated in FIG. 15 may grasp the classification result with almostno movement of the viewpoint.

In addition to coloring the frame 201 or the like in accordance with theclassification result, the medical image processing apparatus 14 may beconfigured such that the frame 201 or the like is colored in the casewhere the classification result is a specific classification such as atumor and the frame 201 or the like is not colored in the case where theclassification result is another classification. Alternatively, themedical image processing apparatus 14 may color the frame 201 or thelike in the case of exceptional determination (described later), or maychange the color in accordance with the degree of reliability in thecase where the display is changed in accordance with the degree ofreliability.

The configuration of changing the color may include a configuration ofchanging the density of the same color. For example, deep red may beused if the endoscopic image 38 is classified into a tumor, and lightred may be used if the endoscopic image 38 is classified into anon-tumor. An object to be colored is not limited to the frame 201 orthe like. A region other than the frames 201 and 207 may be colored.Furthermore, a non-color-based display manner such a display mannerusing a symbol can be used as long as the display manner makes themovement of the viewpoint of the operator less and makes it easier forthe operator to grasp the classification result than the textinformation.

Other Modifications

The display manner of the region image may be changed in accordance withthe classification result. For example, in a case where the endoscopicimage 38 is classified into two classes of a tumor and a non-tumor, thecolor used in the case of the tumor may be changed from the color usedin the case of the non-tumor. In such an example, red may be used if theendoscopic image 38 is classified into the tumor, and blue may be usedif the endoscopic image 38 is classified into the non-tumor.

Alternatively, the display screen may be configured such that theplurality of region image display areas 204 are displayable, and thelocation of the region image 208 may be changed in accordance with theclassification result. For example, in a display screen in which tworegion image display areas 204 vertically displayable, the upper regionimage display area 204 may be used if the endoscopic image 38 isclassified into the tumor, and the lower region image display area 204may be used if the endoscopic image 38 is classified into the non-tumor.The modifications of the display screen described above may assist thevisibility of the operator. The display manner according to theclassification result described in the embodiments is an example of adepicting manner.

Display Examples of Region Image According to Degree of Contribution toClassification

FIG. 16 is an explanatory diagram of a region image to which densityaccording to a degree of contribution is applied. In the display screen200 e illustrated in FIG. 16, the density according to the degree ofcontribution is applied to a classification contribution correspondingregion 210 e in a region image 208 e.

In the classification contribution corresponding region 210 eillustrated in FIG. 16, a central portion 211 a is colored deeper and aperipheral portion 211 b is colored lighter. This indicates that thecentral portion 211 a has a high degree of contribution, and theperipheral portion 211 b has a low degree of contribution. Note that thedegree of contribution may be classified in there or more levels.

FIG. 17 is an explanatory diagram of a region image to which a heat mapaccording to the degree of contribution is applied. In the displayscreen 200 f illustrated in FIG. 17, a heat map according to the degreeof contribution is applied to a classification contributioncorresponding region 210 f in a region image 208 f.

In the classification contribution corresponding region 210 fillustrated in FIG. 17, red is used for the central portion 211 a andblue is used for the peripheral portion 211 b. An intermediate colorthat changes from red to blue is used for an intermediate region 211 cbetween the central portion 211 a and the peripheral portion 211 b.Examples of the intermediate color include orange, yellow, green, and soon.

The classification contribution corresponding region 210 f illustratedin FIG. 17 indicates that the degree of contribution of the centralportion 211 a is high, the degree of contribution decreases from thecentral portion 211 a toward the peripheral portion 211 b, and thedegree of contribution of the peripheral portion 211 b is the lowest.Note that two or more of the modifications described herein may beappropriately combined with one another.

Detailed Description of Classification and Depiction of Region thatContributes to Classification

A specific example of classification of the endoscopic image 38 anddepiction of a region that contributes to the classification will bedescribed next.

Pattern 1

As a pattern 1, an example is presented in which a feature quantity iscalculated from the endoscopic image 38, classification is performed onthe basis of the feature quantity, and a region that contributes to theclassification is depicted. A method based on a support vector machine(SVM) or the like may be used in the classification based on the featurequantity. For example, a blood vessel region is extracted from theendoscopic image 38, and a feature quantity of the extracted bloodvessel region is calculated.

Other methods include a method of performing texture analysis on theendoscopic image 38 and calculating a feature quantity using theanalysis result, a method of calculating a local feature quantity suchas scale-invariant feature transform (SIFT), and so on.

The feature quantity calculated using any of the aforementioned methodsis analyzed in units of regions obtained in the case where the targetimage is divided into a plurality of regions. In this way, the classmembership probability can be calculated in units of regions. Thisconsequently enables depiction processing to be performed for individualregions based on the respective class membership probabilities in unitsof regions. The plurality of regions described in the embodiments are anexample of a plurality of regions set in a captured image.

Pattern 2

As a pattern 2, an example is presented in which information of anintermediate layer of a convolutional neural network is analyzed toclassify the endoscopic image 38 and a region that contributes to theclassification is depicted. Such a method enables classification anddepiction to be processed in parallel.

FIG. 18 is a block diagram illustrating an example of a configuration ofa classification unit that employs a convolutional neural network.Hereinafter, the convolutional neural network may be abbreviated as CNN.The classification unit that employs the convolutional neural networkdescribed in the embodiments is an example of a deep learning device.

A classification unit 300 illustrated in FIG. 18 is an example of theclassification unit 48 illustrated in FIG. 3. The classification unit300 includes a CNN 302, an error calculation unit 304, and a parameterupdating unit 306.

The CNN 302 performs image recognition on the type of a lesion containedin the endoscopic image 38. The CNN 302 has a structure of a pluralityof layers and holds a plurality of weight parameters. The weightparameters of the CNN 302 are updated from the initial values to theoptimum values. This may change an untrained model to a trained model.

The CNN 302 includes an input layer 310, an intermediate layer 312, andan output layer 314. The intermediate layer 312 includes a plurality ofsets of a convolutional layer 320 and a pooling layer 322. Theintermediate layer 312 also includes a fully connected layer 324. Eachlayer is structured such that a plurality of nodes are connected to oneanother by edges.

The endoscopic image 38 serving as a target to be learned is input tothe input layer 310. The intermediate layer 312 extracts features fromthe endoscopic image 38 input thereto from the input layer 310. Theconvolutional layer 320 performs filtering processing on nearby nodes inthe previous layer to acquire a feature map. Note that the filteringprocessing is synonymous with a convolution operation using a filter.

The pooling layer 322 reduces the size of the feature map output fromthe convolutional layer 320 to generate a new feature map. Theconvolutional layer 320 plays a role of extracting features, such asextracting edges, from the endoscopic image 38.

The pooling layer 322 plays a role of providing robustness so that theextracted features are not affected by translation or the like. Notethat the intermediate layer 312 is not limited to the case where theconvolutional layer 320 and the pooling layer 322 constitute a singleset, and the case where the convolutional layers 320 are consecutivelyarranged and a configuration including a normalization layer (notillustrated) are also possible.

The output layer 314 classifies the type of the lesion contained in theendoscopic image 38 on the basis of the features extracted using theintermediate layers 312. The trained CNN 302 may classify, for example,a medical image into two classes, that is, a tumor or a non-tumor. Therecognition result may be obtained as two kinds of scores correspondingto the tumor or the non-tumor.

Any initial values are set for the coefficient and the offset value ofthe filter used in the convolutional layer 320 and the connection weightto the next layer of the fully connected layer 324 in the CNN 302 thathas not been trained yet.

The error calculation unit 304 acquires the recognition result outputfrom the output layer 314 of the CNN 302 and correct answer data 370 forthe endoscopic image 38, and calculates an error therebetween. Examplesof the error calculation method include the softmax cross entropy, thesigmoid, and the like.

The parameter updating unit 306 adjusts the weight parameters of the CNN302 by applying backpropagation based on the error calculated using theerror calculation unit 304. The parameter updating unit 306 repeatedlyperforms the parameter adjustment processing and repeatedly performstraining until the difference between the output of the CNN 302 and thecorrect answer data 370 becomes small.

The classification unit 300 performs training for optimizing eachparameter of the CNN 302 by using a data set of the endoscopic images 38stored in a database (not illustrated) to generate a trained model.

FIG. 19 is a conceptual diagram of the shape transition of the featurequantity in the convolutional neural network. A reference sign 330denotes schematized information. Input data represents information 330 aof the input layer 310. Conv.1 to Conv.5 represent the intermediatelayer 312. FC6 to FC8 represent the fully connected layer 324. In FIG.19, an information amount of the information 330 a of the input layer310 and an information amount of information 330 b of the intermediatelayer 312 are represented as width×height×number of channels.

The convolutional neural network performs final output while aggregatingspatial information of an input image. As illustrated in FIG. 19, theconvolutional neural network aggregates spatial information having awidth and a height as the processing advances from the input layer 310to the intermediate layer 312. Consequently, the spatial information islost in information 330 c of the fully connected layer 324 andthereafter.

That is, analysis of the information 330 b of the intermediate layer 312enables a feature retaining spatial information to be extracted.However, in general, the number of channels of the information 330 b ofthe intermediate layer 312 is larger than that of the information 330 aof the input layer 310.

In the example illustrated in FIG. 19, the number of channels of theinformation 330 a is three in the input layer 310, whereas the number ofchannels of the information 330 b is increased to 384 in Conv. 4 of theintermediate layer 312. Accordingly, how to aggregate the information inthe channel direction to depict the classification contribution regionbecomes an issue. Examples in which information in the channel directionis aggregated to depict the classification contribution region will bedescribed below.

First Example

In general, a channel that greatly contributes to the final output tendsto have a large absolute value. Accordingly, the absolute values ofinformation in the respective channels of the intermediate layer 312 arecompared with each other, and a channel having a larger absolute valueis extracted. The extracted channel is depicted. This enables a regionthat contributes more to the final output to be depicted.

When the channel is extracted, a plurality of channels may be extracted.As an example of the case where a plurality of channels are extracted,there is an example in which a predetermined number of channels areextracted in descending order of the absolute values. When a pluralityof channels are extracted, the plurality of channels may be averaged.

Second Example

Principal component analysis may be performed in the channel directionof the intermediate layers 312 to extract a principal component, and theextracted principal component may be depicted. For example, theextracted principal component can be depicted by reducing the dimensionsof the channels to one dimension. In the second example, sinceinformation of all channels is depicted, more accurate depiction thanthat in the first example is possible.

Third Example

The final output result is a score for each class label of theclassification. The degree of contribution to the score of each classlabel of the classification can be derived using differentiation. Forexample, Gradient-weighted Class Activation Mapping (Grad-CAM) may beused to derive the degree of contribution to the score of each classlabel of the classification.

Let y^(c) denote the score of an arbitrary class c. Let A^(k) denote afeature map of a k-th channel of an arbitrary intermediate layer. LetA^(k) _(ij) denote the values of the coordinates (i, j) of the featuremap A^(k). A map L^(C)Grad-CAM in which the degree of contribution ofthe class c is depicted is obtained by Equation below.

$L_{{Grad} - {CAM}}^{c} = {{ReLU}\left( {\sum\limits_{k}{\alpha_{k}^{c}A^{k}}} \right)}$$\alpha_{k}^{c} = {\frac{1}{Z}{\sum\limits_{i}{\sum\limits_{j}\frac{\partial y^{c}}{\partial A_{ij}^{k}}}}}$$Z = {\sum\limits_{i}{\sum\limits_{j}1}}$ ReLU(x) = max {x, 0}

The map L^(C)Grad-CAM represents a region image depicting theclassification contribution region in which information in the channeldirection is aggregated.

FIG. 20 is a conceptual diagram of depiction based on information of anintermediate layer of a convolutional neural network. The exampleillustrated in FIG. 20 presents an example in which an image to beprocessed 380 including two kinds of animals is classified into oneanimal 381 a or another animal 381 b and a region serving as a basis ofthe classification is depicted.

A first classified image 382 presents an example in which the image tobe processed 380 is classified into the one animal 381 a and a region384 serving as the basis of the classification is depicted. A secondclassified image 386 presents an example in which the image to beprocessed 380 is classified into the other animal 381 b and a region 388serving as the basis of the classification is depicted.

When the pattern 2 is used in classification of the endoscopic image 38,the image to be processed 380 may be replaced with the endoscopic image38, and the two kinds of animals may be replaced with two kinds ofclasses. The two kinds of animals may be replaced with two kinds offeature regions.

Pattern 3

As a pattern 3, an example is presented in which a region is depictedusing a segmentation technique. FIG. 21 is an explanatory diagram of thesegmentation technique. FIG. 21 illustrates an example in which an imageto be classified 390 including a person 391 a and an animal 391 b isclassified into a person region 394, an animal region 396, and abackground region 398. A reference sign 392 denotes a classified image.

With the segmentation technique, a class membership probability isobtained for each region. Each region can be depicted based on themembership probability. Classification can be performed on the image tobe classified 390 on the basis of the regions obtained using thesegmentation technique. For example, a class having the largest area inthe image to be classified 390 among the classes used in classificationmay be set as the classification result.

Advantageous Effects of First Embodiment

With the medical image processing apparatus according to the firstembodiment, the following advantageous effects can be obtained.

[1] The endoscopic image 38 is displayed in the endoscopic image displayarea 202 of the display screen 200 displayed on the monitor device 16.The region image 208 is displayed in the region image display area 204different from the endoscopic image display area 202 of the displayscreen 200. The region image 208 includes the classificationcontribution corresponding region 210 corresponding to theclassification contribution region 209 that contributes toclassification of the endoscopic image 38. This allows an observer tovisually grasp which region in the endoscopic image 38 theclassification unit 48 performs classification on the basis of. Inaddition, the region image 208 may serve as an index of the reliabilityof the classification result.

[2] The classification result is displayed on the monitor device 16.This allows the observer to visually recognize the classification resultof the endoscopic image 38.

[3] A membership degree for each class is derived as the classificationresult. The membership degree of each class is displayed on the monitordevice 16. As the membership degree, a membership probability for eachclass or a score for each class may be used. This allows the observer tovisually recognize the membership degree for each class.

[4] A display manner of the region image 208 may be changed inaccordance with the classification result. This may improve thevisibility of the classification result.

Description of Medical Image Processing Apparatus According to SecondEmbodiment

A medical image processing apparatus according to a second embodimentwill be described next. In the medical image processing apparatusaccording to the second embodiment, exception determination is added toclassification of the endoscopic image 38. That is, a lesion region ofthe endoscopic image 38 is depicted using the pattern 1, the pattern 2,and the pattern 3 described in the first embodiment, and then the regionimage is analyzed to make exception determination.

For example, if the area of the lesion region detected from theendoscopic image 38 is equal to or less than a predetermined thresholdvalue, the classification result may be set as undetectable. If thereare a plurality of lesion regions detected from the endoscopic image 38and the areas of all the lesion regions are equal to or greater than apredetermined threshold value, the classification result may be set asundeterminable.

Note that the medical image processing apparatus according to the secondembodiment employs substantially the same hardware and functional blocksas those of the medical image processing apparatus 14 according to thefirst embodiment. Thus, description of the hardware and functionalblocks of the medical image processing apparatus according to the secondembodiment will be omitted.

Display Screen Displayed in Case of Being Undetectable

FIG. 22 is an explanatory diagram of a display screen indicating theclassification result of “undetectable”. In a display screen 400illustrated in FIG. 22, undetectable indicating the classificationresult of “undetectable” is displayed as text information 412. When thetext information 412 representing undetectable is displayed, aclassification contribution corresponding region 410 is not displayed ina region image 408. Note that FIG. 22 schematically illustrates, using asolid line, a lesion region 407 that is undetectable. The lesion region407 that is detectable is schematically illustrated using a two-dotchain line.

A configuration in which the text information representing theclassification result is overwritten with text information representingundetectable may be used as the text information 412. The display screen400 may employ a configuration indicating that the lesion isundetectable, separately from the text information indicating theclassification result.

The classification unit 48 illustrated in FIG. 3 may analyze the regionimage 408 to quantify the degree of reliability of the classification.The display control unit 44 may display a numerical value such as ascore representing the degree of reliability of the classification inthe display screen 400. Note that a reference sign 402 illustrated inFIG. 22 denotes the endoscopic image display area. A reference sign 404denotes the region image display area. A reference sign 406 denotes theclassification result display area.

Display Screen Displayed in Case of Being Undeterminable

FIG. 23 is an explanatory diagram of a display screen indicating theclassification result of “undeterminable”. In a display screen 400 aillustrated in FIG. 23, undeterminable indicating the classificationresult of “undeterminable” is displayed as text information 412 a. Whenthe text information 412 a representing undeterminable is displayed, aclassification contribution corresponding region 410 a indicating theundeterminable state is displayed in a region image 408 a.

Examples of the classification contribution corresponding region 410 aindicating the undeterminable state include an example in which twotypes of display manners corresponding to different classificationscoexist. FIG. 23 illustrates the classification contributioncorresponding region 410 a in which two colors coexist in the case wherethe color is changed in accordance with the classification. Note that areference sign 409 in FIG. 23 denotes the classification contributionregion.

Advantageous Effects of Second Embodiment

With the medical image processing apparatus according to the secondembodiment, the following advantageous effects can be obtained.

[1] When a lesion in the endoscopic image 38 is undetectable, the textinformation 412 indicating that the lesion is undetectable is displayed.When a lesion in the endoscopic image 38 is undeterminable, the textinformation 412 indicating that the lesion is undeterminable isdisplayed. This allows the operator to grasp the inappropriateclassification.

[2] The degree of reliability of classification is calculated on thebasis of the region image. The degree of reliability is displayed in thedisplay screen. This enables the degree of reliability of classificationto be recognized.

Description of Medical Image Processing Apparatus According to ThirdEmbodiment

FIG. 24 is a functional block diagram of a medical image processingapparatus according to a third embodiment. A medical image processingapparatus 14 a illustrated in FIG. 24 includes a storage instructionacquisition unit 41, compared to the medical image processing apparatus14 illustrated in FIG. 3. The storage instruction acquisition unit 41acquires an instruction to store the endoscopic image 38, transmittedfrom the processor device 12 illustrated in FIG. 1. The storageinstruction acquisition unit 41 may acquire the instruction to store theendoscopic image 38 from the endoscope 10.

In response to the storage instruction acquisition unit 41 acquiring theinstruction to store the endoscopic image 38, the endoscopic image 38acquired using the image acquisition unit 40 is stored in the endoscopicimage storage unit 46 a in association with at least any of theclassification result, the region image, the exception determinationresult, or the degree of reliability.

The medical image processing apparatus 14 a may combine theclassification result and the like with the endoscopic image 38 andstore the combined result, or may separately store the classificationresult and the like and the endoscopic image 38. In response to theoperator operating an operation button (not illustrated) or the like,the instruction to store the endoscopic image 38 may be transmitted fromthe processor device 12. Alternatively, the instruction to store theendoscopic image 38 may be automatically transmitted from the processordevice 12 on the basis of the classification result or the like. Whenthe degree of reliability is equal to or higher than a predeterminedthreshold value, the processor device 12 may determine that the medicalimage processing apparatus 14 a has obtained an appropriateclassification result and automatically transmit the instruction tostore the endoscopic image 38.

Advantageous Effects of Third Embodiment

With the medical image processing apparatus according to the thirdembodiment, the following advantageous effects can be obtained.

[1] When a user creates a report after an endoscopic examination, theuser may check whether the classification result or the like isappropriate.

[2] At the time of re-examination, examination results of the precedingexaminations may be referred to.

Modifications of Endoscope System Modification of Processor Device

The processor device 12 may have the functions of the medical imageprocessing apparatus 14. That is, the processor device 12 and themedical image processing apparatus 14 may be integrated together. Insuch an embodiment, the display device 13 may also serve as the monitordevice 16. The processor device 12 may include a connection terminal towhich the input device 15 is connected.

Modification of Illumination Light

One example of the medical image acquirable using the endoscope system 9according to the present embodiments is a normal-light image acquired byradiating light in a white range or light in a plurality of wavelengthranges as the light in the white range.

Another example of the medical image acquirable using the endoscopesystem 9 according to the present embodiments is an image acquired byradiating light in a specific wavelength range. A range narrower thanthe white range may be used as the specific wavelength range. Thefollowing modifications may be employed.

First Example

A first example of the specific wavelength range is a blue range or agreen range in a visible range. The wavelength range of the firstexample includes a wavelength range of 390 nm or more and 450 nm or lessor a wavelength range of 530 nm or more and 550 nm or less, and thelight of the first example has a peak wavelength in the wavelength rangeof 390 nm or more and 450 nm or less or the wavelength range of 530 nmor more and 550 nm or less.

Second Example

A second example of the specific wavelength range is a red range in thevisible range. The wavelength range of the second example includes awavelength range of 585 nm or more and 615 nm or less or a wavelengthrange of 610 nm or more and 730 nm or less, and the light of the secondexample has a peak wavelength in the wavelength range of 585 nm or moreand 615 nm or less or the wavelength range of 610 nm or more and 730 nmor less.

Third Example

A third example of the specific wavelength range includes a wavelengthrange in which an absorption coefficient is different betweenoxyhemoglobin and deoxyhemoglobin, and the light of the third examplehas a peak wavelength in the wavelength range in which the absorptioncoefficient is different between oxyhemoglobin and deoxyhemoglobin. Thewavelength range of this third example includes a wavelength range of400±10 nm, a wavelength range of 440±10 nm, a wavelength range of 470±10nm, or a wavelength range of 600 nm or more and 750 nm or less, and thelight of the third example has a peak wavelength in the wavelength rangeof 400±10 nm, the wavelength range of 440±10 nm, the wavelength range of470±10 nm, or the wavelength range of 600 nm or more and 750 nm or less.

Fourth Example

A fourth example of the specific wavelength range is a wavelength rangeof excitation light that is used to observe fluorescence emitted by afluorescent substance in a living body and excites this fluorescentsubstance. For example, the specific wavelength range of the fourthexample is a wavelength range of 390 nm or more and 470 nm or less. Notethat observation of fluorescence may be referred to as fluorescenceobservation.

Fifth Example

A fifth example of the specific wavelength range is a wavelength rangeof infrared light. The wavelength range of this fifth example includes awavelength range of 790 nm or more and 820 nm or less or a wavelengthrange of 905 nm or more and 970 nm or less, and the light of the fifthexample has a peak wavelength in the wavelength range of 790 nm or moreand 820 nm or less or the wavelength range of 905 nm or more and 970 nmor less.

Generation Example of Special-Light Image

The processor device 12 may generate a special-light image havinginformation in the specific wavelength range on the basis of anormal-light image obtained through imaging using white light. Note thatthe term “generation” used herein includes “acquisition”. In this case,the processor device 12 functions as a special-light image acquisitionunit. The processor device 12 obtains a signal of the specificwavelength range by performing calculation based on color information ofred, green, and blue or color information of cyan, magenta, and yellowincluded in the normal-light image.

Note that red, green, and blue are sometimes referred to as RGB. Inaddition, cyan, magenta, and yellow are sometimes referred to as CMY.

Generation Example of Feature-Quantity Image

As the medical image, a feature-quantity image may be generated by usingan operation based on at least any of a normal-light image obtained byradiating light in the white range or light in a plurality of wavelengthranges as the light in the white range or a special-light image obtainedby radiating light in the specific wavelength range.

Application Example to Program for Causing Computer to Function as ImageProcessing Apparatus

The above-described image processing method can be configured as aprogram that implements functions corresponding to respective steps ofthe image processing method using a computer. For example, a programthat causes a computer to implement an endoscopic image acquisitionfunction, an image processing function, a display signal transmissionfunction, and a storage function may be configured. The image processingfunctions may include a classification function and a region imagegeneration function.

A program that causes a computer to implement the above-described imageprocessing function may be stored on a computer-readable informationstorage medium which is a non-transitory tangible information storagemedium, and the program may be provided using the information storagemedium.

In addition, instead of the configuration in which the program is storedon a non-transitory information storage medium and is provided, aconfiguration in which a program signal is provided via a network may beemployed.

Combination of Embodiments, Modifications, etc.

The constituent elements described in the embodiments above and theconstituent elements described in the modifications can be appropriatelyused in combination, and some of the constituent elements can bereplaced.

Application Examples to Other Devices

In the embodiments described above, an endoscopic image is used as anexample of a medical image. However, automatic classification describedin the present embodiments is also applicable to a medical imageacquired using a CT apparatus, an MRI apparatus, an X-ray imagingapparatus, or the like. The medical image described in the embodimentsis an example of a captured image.

In the embodiments of the present invention described above, theconstituent elements can be appropriately changed, added, or deletedwithin a scope not departing from the gist of the present invention. Thepresent invention is not limited to the embodiments described above, andvarious modifications can be made by a person having the ordinary skillin the art within the technical sprit of the present invention.

REFERENCE SIGNS LIST

9 endoscope system

10 endoscope

11 light source device

12 processor device

13 display device

14 medical image processing apparatus

14 a medical image processing apparatus

15 input device

16 monitor device

20 insertion section

21 operation section

22 universal cord

25 soft part

26 bending part

27 tip part

27 a tip surface

28 imaging element

29 bending operation knob

30 air/water supply button

31 suction button

32 still image capturing instruction part

33 treatment tool introduction port

35 light guide

36 signal cable

37 a connector

37 b connector

38 endoscopic image

38 a moving image

38 b frame image

39 still image

40 image acquisition unit

41 storage instruction acquisition unit

42 image processing unit

44 display control unit

46 storage unit

46 a endoscopic image storage unit

46 b classification storage unit

46 c region image storage unit

48 classification unit

50 region image generation unit

120 control unit

122 memory

124 storage device

126 network controller

128 power supply device

130 display controller

132 input/output interface

134 input controller

136 bus

140 network

200 display screen

200 a display screen

200 b display screen

200 c display screen

200 d display screen

200 e display screen

201 frame

202 endoscopic image display area

204 region image display area

206 classification result display area

206 a classification result display area

207 frame

208 region image

208 a region image

208 b region image

208 c region image

208 d region image

208 e region image

208 f region image

208 g region image

208 h region image

209 classification contribution region

209 a first classification contribution region

209 b second classification contribution region

210 classification contribution corresponding region

210 a region to be set as classification contribution correspondingregion

210 b region to be set as classification contribution correspondingregion

210 e classification contribution corresponding region

210 f classification contribution corresponding region

210 g first classification contribution corresponding region

210 h second classification contribution corresponding region

211 a central portion

211 b peripheral portion

211 c intermediate region

212 text information

212 a text information

212 b text information

212 c text information

212 d text information

212 e first text information

212 f second text information

213 a first lesion

213 b second lesion

220 comparative screen

222 tumor

230 region

300 classification unit

302 CNN

304 error calculation unit

306 parameter updating unit

310 input layer

312 intermediate layers

314 output layer

320 convolutional layer

322 pooling layer

324 fully connected layer

330 information

330 a information of input layer

330 b information of intermediate layer

330 c information of fully connected layer

370 correct answer data

380 image to be processed

381 a one animal

381 b another (other) animal

382 first classified image

384 region serving as basis of classification

386 second classified image

388 region serving as basis of classification

390 image to be classified

391 a person

391 b animal

392 classified image

394 person region

396 animal region

400 display screen

400 a display screen

402 endoscopic image display area

404 region image display area

406 classification result display area

408 region image

410 classification contribution corresponding region

412 text information

412 a text information

S10 to S22 steps of medical image processing method

What is claimed is:
 1. A medical image processing apparatus comprising:an image acquisition unit that acquires a captured image generatedthrough imaging of a living body; a classification unit that classifiesthe captured image into two or more classes; an image generation unitthat generates a region image depicting a location of a region thatcontributes to classification performed using the classification unit,in the captured image subjected to the classification performed usingthe classification unit; and a display signal transmission unit thattransmits, to a display device, a display signal representing thecaptured image, a classification result derived using the classificationunit, and the region image, wherein the display signal transmission unittransmits, to the display device, a display signal for displaying theregion image separately from the captured image.
 2. The medical imageprocessing apparatus according to claim 1, wherein the image generationunit changes a depicting manner of the region image in accordance withthe classification result of the captured image.
 3. The medical imageprocessing apparatus according to claim 1, wherein the classificationunit classifies the captured image on the basis of a feature quantityacquired from the captured image, and the image generation unitgenerates the region image on the basis of the feature quantity.
 4. Themedical image processing apparatus according to claim 1, wherein theclassification unit employs a deep learning device that has beentrained, and the image generation unit generates the region image on thebasis of information of an intermediate layer of the deep learningdevice.
 5. The medical image processing apparatus according to claim 1,wherein the classification unit calculates, for each of a plurality ofregions set in the captured image, membership degrees for the classes,and classifies the captured image on the basis of the membershipdegrees.
 6. The medical image processing apparatus according to claim 5,wherein the image generation unit generates the region image on thebasis of the membership degrees.
 7. The medical image processingapparatus according to claim 1, wherein the classification unit performsexception determination for the classification based on the regionimage, and the display signal transmission unit transmits, to thedisplay device, a display signal representing a result of the exceptiondetermination performed using the classification unit.
 8. The medicalimage processing apparatus according to claim 7, wherein theclassification unit calculates, on the basis of the region image, adegree of reliability of the classification result derived using theclassification unit, and the display signal transmission unit transmits,to the display device, a display signal representing the degree ofreliability.
 9. The medical image processing apparatus according toclaim 8, further comprising: a storage instruction acquisition unit thatacquires an instruction to store the captured image; and a storage unitthat stores the captured image, wherein the storage unit associates, instoring the captured image in the storage unit in accordance with theinstruction to store the captured image, at least any of theclassification result, the result of the exception determination, or thedegree of reliability of the classification result with the capturedimage.
 10. The medical image processing apparatus according to claim 1,wherein the display signal transmission unit transmits to the displaydevice, a display signal representing text information of theclassification result.
 11. A medical image processing system comprising:an image acquisition unit that acquires a captured image generatedthrough imaging of a living body; a classification unit that classifiesthe captured image into two or more classes; an image generation unitthat generates a region image depicting a location of a region thatcontributes to classification performed using the classification unit,in the captured image subjected to the classification performed usingthe classification unit; a display device that displays the capturedimage and the region image; and a display signal transmission unit thattransmits, to a display device, a display signal representing thecaptured image and the region image, wherein the display signaltransmission unit transmits, to the display device, a display signal fordisplaying the region image separately from the captured image.
 12. Amedical image processing method comprising: an image acquisition step ofacquiring a captured image generated through imaging of a living body; aclassification step of classifying the captured image into two or moreclasses; an image generation step of generating a region image depictinga location of a region that contributes to classification performed inthe classification step, in the captured image subjected to theclassification performed in the classification step; and a displaysignal transmission step of transmitting, to a display device, a displaysignal representing the captured image, a classification result derivedin the classification step, and the region image, wherein the displaysignal transmission step transmits, to the display device, a displaysignal for displaying the region image separately from the capturedimage.
 13. A non-transitory computer-readable storage medium storingcommands that, when read by a computer, cause the computer to execute:an image acquisition function that acquires a captured image generatedthrough imaging of a living body; a classification function thatclassifies the captured image into two or more classes; an imagegeneration function that generates a region image depicting a locationof a region that contributes to classification performed using theclassification function, in the captured image subjected to theclassification performed using the classification function; and adisplay signal transmission function that transmits, to a displaydevice, a display signal representing the captured image, aclassification result derived using the classification function, and theregion image, the display signal transmission function being configuredto transmit, to the display device, a display signal for displaying theregion image separately from the captured image.